Graphical 3D Curve Rendering Data Visualization Tools and Primitives for Tabular Data and Spreadsheets

ABSTRACT

Graphical tools and primitives are provided for visualization of graphical curve rendered in  3 D from tabular data such as in a spreadsheet. The invention can be used for simple business analysis as well as advanced engineering and science. Visualizations of a one-dimensional curve are rendered from provided data according to position, orientation, and viewpoint, some or all of which can be controlled by a user interface. The visualization can be displayed together with an electronic spreadsheet or other tabular data representation. The curve can be suspended over a region of the displayed tabular data or otherwise displayed. The visually rendered curve can comprise symbols or varying colors, thicknesses parameterized textures, or parameterized glyphs representing data, interactively controlled filtered data, or stochastically processed data. The curve can be suspended over a region of displayed tabular data. A visual height tool and visual links to tabular data items can be included.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 12/875,115filed on Sep. 2, 2010, the contents of which are incorporated byreference, which claims benefit of priority of U.S. ProvisionalApplication No. 61/239,349 filed on Sep. 2, 2009, incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to data visualization and in particular itsimplementation by and application to conventional spreadsheets anddisplays of tabular data.

2. Background of the Invention

Interactive electronic spreadsheets such as the 1979 VisiCalc numericalspreadsheet (http://www.danbricklin.com/visicalc.htm), Supercalcnumerical spreadsheet (1980), Microsoft Multiplan™ 91982), Lotus 1-2-3™(1983), the AppleWorks™ spreadsheet module (1984), Microsoft Excel™(1985), OpenOffice.org™ Calc, Google's Google Spreadsheets, etc.,present a tabular presentation of data and calculation formula flow inarrangements that have proven to be very useful for data analysis.Interactive electronic spreadsheets typically include plotting routinesthat create graphical representation of selected data points. Theseplots can be used to provide visual representations of data andmathematical functions. In a sense, these interactive electronicspreadsheet functions provide a form of data visualization based on datain the interactive electronic spreadsheet.

Further, interactive electronic spreadsheets present numerical, textual,and symbolic data to be presented in a tabular array of cells, with thenumerical, textual, and symbolic data within the cells provided invarious font styles and colors, and with various background colors andborder styles of the associated cell. Using interactive electronicspreadsheet functions known as “conditional formatting” these fontstyles and colors, cell background colors, and cell border styles can becontrolled by values of data according to hand-specified conditionaltests performed on data within one or more specified cells. Suchhand-specified conditional tests are entered through interactiveelectronic spreadsheet GUIs (Graphical User Interfaces), VBA (VisualBasic for Applications), APIs (Application programmer interfaces), etc.These interactive electronic spreadsheet functions provide another formof data visualization based on data in the interactive electronicspreadsheet.

These visual data representation successes of the interactive electronicspreadsheet paradigm suggest additional opportunities for providing andcombining additional data visualization capabilities with thewell-established functionality and embedded deployment of interactiveelectronic spreadsheet software. The present invention provides severalsuch additional data visualization capabilities to traditionalinteractive electronic spreadsheets. The visual effects provided by theinvention can be automatically varied over a range responsive to valuesof data or formulas according to mathematical functions rather thanconditional tests found in interactive electronic spreadsheet products.

Unlike the logical-test structure inherent in “conditional formatting”,the invention provides for use of numerical values directly from tabulardata or as calculated by hidden mathematical function, compositions ofmathematical functions, or traditional spreadsheet formulas to easilyprovide rich detailed control of useful parameterized visual effectsrendered in the context of conventional interactive electronicspreadsheet and tabular data display.

The visual effects can include variation of background color or textureor border color, thickness, grouping scope of cells envelopingcharacters conveying data, as well as font color, type, embellishment,size, format, location, decimal places, or supplemental symbols symbolicor image element rendered within the cell of characters conveying data.Additionally, the invention provides for data-driven 3D plots renderedin the context of or projection from interactive electronic spreadsheetor tabular data. The invention provides for these visualizationoperations to be generated by an algorithm directly or indirectly incommunication with data or parts of a program rendering an interactiveelectronic spreadsheet or visually displayed data table. The inventionalso provides for various optional additional functions, for example:

-   -   Data-driven automatic “live” sorting of rows and columns;    -   Incorporation of externally-provided stored data and live data        feeds;    -   Real-time response to live input data, interactive data entry,        or visualization manipulation via interactive user interfaces;    -   Collaboration capabilities;    -   Web-access capabilities;    -   Meaningful data sonification capabilities, including those        involving multichannel timbre modulation.

Because many aspects of the invention are described in terms of, orpertain to interactive electronic spreadsheet programs, the shorthand“spreadsheet” will be understood to mean interactive electronicspreadsheet programs.

SUMMARY OF THE INVENTION

The invention provides for a method to visually enhance the display oftabular data of an electronic spreadsheet comprising cells arranged inrows and columns. At least one data value is used to compute at leastone visual parameter, and each visual parameter is a number within arange of numbers and is determined according to a numerically calculatedmathematical function. A visual effect of the at least one data value iscontrolled according to the at least one visual parameter, and thevisual effect is applied to the display of the at least one data valueto visually enhance the display of tabular data of the electronicspreadsheet.

Various aspects and features of the invention are described below. Theseaspects and features may be found either singly or in combination in theinvention, and the descriptions of the various aspects and features areintended to be illustrative and not to limit the scope of the invention.

The visual effects can be automatically varied over a range responsiveto data values or formulas according to mathematical functions ratherthan conditional tests found in interactive electronic spreadsheetproducts. The visual parameters of a data or cell presentation attributemay be varied over a pre-defined range (for example, between 0 and 1,between 0 and 100, between −1 and +1, etc).

Various attributes of the font in which the tabular data is displayedcan be controlled. For example, one or more of the font color, the fonttype or style, font embellishments, and the font size may be controlledby the visual parameters. If the tabular data is numeric data, one ormore of the number format, the number of decimal places, and the methodand degree of rounding may also be controlled by the visual parameters.

The background of the cell in which tabular data is displayed may alsobe visually enhanced by the visual parameters. For example, one or moreof the background color and background texture may be controlled by thevisual parameters.

Further, one or more of border color and border line thickness may becontrolled by the visual parameters. The border may enclose a singlespreadsheet cell displaying tabular data or may enclose a plurality ofcontiguous cells displaying tabular data.

According to the visual parameters, a symbol may be appended to thedisplayed tabular data, or an image or a data symbol may beindependently displayed instead of the tabular data in a spreadsheetcell.

While each of the above aspects and features of the invention may beapplied to individual spreadsheet cells, these aspects and features maybe applied to rows and columns containing tabular data.

Other aspects and features of the invention include:

-   -   visual parameters are generated by an algorithm in communication        with the spreadsheet program, or from stored data;    -   storage and recall of sessions.    -   the support of data-driven automatic “live” rows and columns        sorting;    -   live data feeds;    -   externally provided data;    -   data sonification;    -   web access;    -   collaboration with other users;    -   3D capabilities; and    -   at least one user interface that providing the selection of        function blocks and the general connectivity among them to be        specified using a drawing tool and a palette of function blocks,        wherein clicking on each function blocks would cause dialog        windows to be displayed for setting parameters by means of        typed-in values, sliders, mouse manipulation, or advanced        high-dimension user interface devices.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of the presentinvention will become more apparent upon consideration of the followingdescription of preferred embodiments taken in conjunction with theaccompanying drawing figures. The accompanying figures are examples ofthe various aspects and features of the present invention and are notlimiting either individually or in combination.

FIG. 1 provides a demography of ways in which data visualization andspreadsheets can be integrated together.

FIG. 2A depicts a region of a spreadsheet or table of tabular data.

FIG. 2B depicts a cell and its immediate neighborhood within aspreadsheet or table of tabular data.

FIG. 3 depicts of data and cell presentation attributes serving ascandidates for visualization parameters within a spreadsheet display.

FIGS. 4A-4E illustrate several application settings where the inventionis well-suited to add value.

FIG. 5 illustrates an arrangement of selected more general aspects ofthe invention that are not restricted to spreadsheet visualization.

FIG. 6 illustrates an adaptation of the arrangement of FIG. 5 to anenvironment supporting collaboration features.

FIG. 7 depicts a topological interconnection of data flow paths linkingvarious elements depicted in FIG. 5.

FIGS. 8A-8C depict approaches for mapping a data value lying within apre-defined range to a value within a pre-defined range for aparameterized data or cell presentation attribute.

FIG. 9 depicts a more general view and organization of pre-visualizationoperations provided for by the invention.

FIG. 10 depicts an embodiment wherein a selected metaphor is used toautomatically generate parameter assignments and graphics renderingoperations.

FIG. 11A illustrates a user interface that permits users to select thegeometric element boundary of a captured or rendered tabular matrix oftabular matrix data elements or cells within a spreadsheet. FIG. 11Billustrates a user interface that permits users to select arrayoperations to be performed.

FIGS. 12A-12G depict transformations of graphics objects via arrayoperations as provided for by the invention.

FIG. 13 depicts a spreadsheet architecture and information flow as itmight be used in a data visualization setting.

FIGS. 14A-14D depict various types of architectural approaches forcombining and integrating spreadsheet programs, visualization operationsprovided by the invention, and a data set used by each.

FIG. 15 depicts an embodiment of an augmentation of a spreadsheetprogram with additional spreadsheet features as provided for by theinvention.

FIGS. 16A-16B depict ways in which one or more user interface inputdevices (such as a mouse, trackball, touchpad, etc.) can be managed soas to control graphical user interfaces of a plural software applicationin the context of the invention.

FIGS. 17A-17F illustrate the six independently adjustable degrees offreedom of touch from a single finger that can be simultaneouslymeasured by the HDTP technology.

FIG. 18 suggests general ways in which two or more of theseindependently adjustable degrees of freedom can be adjusted at once.

FIG. 19 demonstrates a few two-finger multi-touch postures and/orgestures from the many that can be readily recognized by HTDPtechnology.

FIG. 20 shows how raw measurements of the six quantities of FIGS.17A-17F, together with shape recognition for distinguishing contact withvarious parts of the hand and the touchpad, can be used to create a richinformation flux of parameters, rates, and symbols.

FIG. 21 shows an approach for incorporating posture recognition, gesturerecognition, state machines, and parsers to create an even richerhuman/machine tactile interface system capable of incorporating syntaxand grammars.

FIG. 22 depicts a user interface input arrangement incorporating one ormore HDTPs that provides user interface input event and quantity routingof the type described earlier in conjunction with FIGS. 16A-16B.

FIGS. 23A-23L depict a number of arrangements employing the HDTPtechnology suitable for use with visualization environments.

FIGS. 24A and 24B illustrate examples of scroll-wheel mice provided withan additional scroll-wheel.

FIGS. 25A-25C illustrate a single trackball incorporated into the backof a conventional computer mouse.

FIGS. 26A-26E illustrate two trackballs incorporated into the back of aconventional computer mouse.

FIG. 27 depicts an arrangement wherein the HDTP or alternatives such asadvanced mice can interface with browser-based applications via abrowser plug-in.

FIG. 28 depicts an implementation for rendering visualization in abrowser.

FIG. 29 depicts a web-based implementation of a visualizationenvironment leveraging browser arrangements such as that depicted inFIGS. 27-28, their variations, alternatives, etc.

FIG. 30A depicts a client-side collaboration implementation wherein ageneral purpose collaboration tool can be used to share thevisualization environment session running on one computer with one ormore other computers.

FIG. 30B depicts a server-side collaboration implementation wherein ageneral purpose or specialized server-side application sharingenvironment is used to share a server-based interactive datavisualization application.

FIG. 30C depicts a server-side collaboration implementation comprising aserver-based multi-user interactive data visualization application. FIG.30D depicts an interface between an HDTP and a browser which directsadditional GUI parameters via a browser plug-in according to an exampleimplementation. FIG. 30E depicts an interface between an HDTP and abrowser, which directs additional GUI parameters through other networkchannels, according to an example implementation. FIG. 30F depicts aninterface between an HDTP and a browser, which interfaces all parametersto the browser directly, according to an example implementation.

FIG. 31 illustrates a general framework for data sonification wherein aparameterized metaphor is used to manage parameter assignment and soundrendering and a parameterized data indexing operation.

FIG. 32 depicts an arrangement wherein the hue of a color can be variedthrough natural colors of red through violet for the bulk of the [0,1]range and a range of artificial colors, such as the purple throughmagenta continuum, are appended for the remaining portion of the [0,1]range.

FIG. 33 illustrates an arrangement wherein line width, line dashingperiod, line dashing duty-cycle, and line dashing sub-duty-cycle can bedetermined by a uniform parameter in the range of [0,1].

FIG. 34 illustrates an arrangement wherein stipple line width, stipplegap, and stipple angle can be determined by a uniform parameter in therange of [0,1].

FIG. 35 depicts an arrangement wherein at least one border renderingoption is to be determined by a uniform parameter in the range of [0,1].The border can be the border of a spreadsheet cell as depicted, or canbe the border of another visually rendered graphical object.

FIG. 36A depicts an arrangement wherein a boldness parameter is underthe control of uniform parameters in the range of [0,1]. FIG. 36Bdepicts an arrangement wherein an italics angle parameter is under thecontrol of uniform parameters in the range of [0,1]. FIG. 36C depicts anarrangement wherein an font family interpolation parameter is under thecontrol of uniform parameters in the range of [0,1].

FIG. 37A depicts an arrangement wherein exemplary vertical andhorizontal dilation/contract parameters are each respectively under thecontrol of uniform parameters in the range of [0,1]. FIG. 37B depicts anarrangement wherein font underline line width and font underlineseparation gap parameters are each respectively under the control ofuniform parameters in the range of [0,1].

FIG. 38 depicts an attribute control panel wherein there arepre-selected values and a value which can be customized.

FIG. 39 depicts an arrangement wherein a change in color of a font canbe specified for a cell, a group of cells, or globally across an entireworksheet.

FIG. 40 depicts an interface that determines a font type of thespecified cell(s).

FIG. 41 depicts an interface that determines the embellishment of a fontin rendering some data element within the spreadsheet or data table.

FIG. 42 depicts an interface to control the size of a font to bedisplayed.

FIG. 43 depicts an wherein data in a cells in a selected column can havesymbol appended to data within the cells of the column.

FIG. 44 depicts an example use of controlling the background texture orstipples of cells as a function of data values in a spreadsheet or otherform of tabular data.

FIG. 45 depicts an interface that can be used to specify the backgroundtexture or stipples to be applied to a cell or a group of cells.

FIG. 46 depicts an interface that can be used to specify the position oftext being displayed in a cell.

FIG. 47 depicts an interface that can be used to specify the rotation oftext being displayed combined with the vertical or horizontal placementof the text.

FIG. 48 depicts a visualization wherein an attribute of a column area ofdata elements is varied as a group.

FIG. 49 depicts a visualization wherein the shading of a of a row areaof data elements is varied as a group as a function of a data value andalso wherein the font boldness of a column area of data elements isvaried as a group as a function of another data value.

FIG. 50 illustrates a priority job queue model tabular-datavisualization wherein the row can automatically reorder among themselvesas underlying priority vales change.

FIG. 51A illustrates a basic map-based visualization example generatedfrom previous data. FIG. 51B illustrates the visualization of FIG. 51Aaugmented with additional spatially located symbols or parameterizedglyphs.

FIG. 52 depicts a 2D-surface representing a 3D data plot of at least3-dimensional data rendered above a planar array of associated tabulardata or spreadsheet, these separated by a vertical gap within a 3Dvisual field as provided for by the invention.

FIG. 53 depicts with dashed lines a few ways in which the observationviewpoint can be moved with respect to these abstract objects.

FIG. 54 depicts a translation and rotation of an exemplary 3D data plotwithin the 3D visual field.

FIG. 55A depicts an arrangement wherein data plotted in a 2D-surfacerepresenting a 3D data plot within the 3D visual field can directly echodata displayed in a planar array of associated tabular data orspreadsheet and/or can originate from another set of tabular data orspreadsheet region and/or can originate from other data.

FIG. 55B depicts an arrangement wherein processing of data viamathematical transformations, statistical processing, signal processing,etc. is made prior to creation of the 2D-surface representing at least3-dimensional data.

FIG. 56 illustrates a height measuring visual in its use with 2D-surfacerepresenting at least 3-dimensional data and useful for representing atleast 4-dimensional data.

FIG. 57 depicts a slicing function here used to provide a (planar-slice)level set curve.

FIG. 58A illustrates two 2D-surfaces, each representing at least3-dimensional data. FIG. 58B illustrates the interactively createdintersection of the two exemplary 2D-surfaces and the resultantintersection curve.

FIG. 59 depicts an arrangement wherein a curve can be generated bytabular data and be suspended over tabular data used to create it orotherwise associated with the curve, and can be visually linked to ageometric rendering of tabular data values.

FIG. 60 depicts an arrangement wherein a curve can be given a thicknessand or color under the control of data values.

FIG. 61 depicts an arrangement wherein the thickness of a curve can bevaried as a function of one or more data values.

FIG. 62 depicts a curve intersection tool that can be used as anumerical solution operator for plotted, interpolated, and processeddata.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawing figures which form a part hereof, and which show by way ofillustration specific embodiments of the invention. It is to beunderstood by those of ordinary skill in this technological field thatother embodiments can be utilized, and structural, electrical, as wellas procedural changes can be made without departing from the scope ofthe present invention. The aspects and features described herein may beused singly or in combination unless specifically stated otherwise.

The visual data representation successes of spreadsheets suggestadditional opportunities for providing and combining additional datavisualization capabilities with the well-established functionality andembedded deployment of spreadsheet software. FIG. 1 provides an exampleof ways in which data visualization and spreadsheets can be integrated.The present invention provides several additional data visualizationcapabilities to traditional spreadsheets.

First, traditional spreadsheets present data in a tabular formleveraging row, column, and sheet organization. Traditional spreadsheetstypically include plotting routines that create graphical representationof selected data points. These plots can be used to provide visualrepresentations of data and mathematical functions. These spreadsheetfunctions provide a base-level form of data visualization based on datain the spreadsheet, as represented in rows A and B of FIG. 1.

Additional subsequent work has been done to employ data visualizationcharacterizing internals of spreadsheet structure such as informationflow, cell formula dependences, and semantic classification. These donot provide visualization of data per se, but can be very useful inunderstanding the data handling of a given spreadsheet. This isrepresented in rows C and D of FIG. 1.

Further subsequent work has been done to supplement basic spreadsheetplotting utilities providing richer data plotting capabilities (forexample, utilizing 3D graphics). These capabilities enhance the possibledata visualization based on data in the spreadsheet. This is representedin row E of FIG. 1.

Yet other subsequent work has been done adopting the interactive formatand metaphor of a spreadsheet for use in a tabular presentation ofcomplex data visualization renderings. This is represented in row F ofFIG. 1. One aspect of the invention focuses on adapting andsignificantly expanding this capability. This is represented in row G ofFIG. 1.

Still other subsequent work has been done superimposing a third (height)dimension atop the 2-dimensional tabular data layout of an interactiveelectronic spreadsheet. The third dimension can be used to rendercolor-coded line and surface plots, and the resulting 3D graphics can beviewed from various virtual observation points for inspection. This isrepresented in row H of FIG. 1. Another aspect of the invention focuseson adapting and expanding this capability.

Traditional spreadsheets also present numerical, textual, and symbolicdata in a tabular array of cells, with the numerical, textual, and/orsymbolic data within the cells provided in various font styles andcolors, and with various background colors and border styles of theassociated cell. Using spreadsheet functions known as “conditionalformatting” these font styles and colors, cell background colors, andcell border styles can be controlled by values of data according tohand-specified (through spreadsheet GUI, VBA, APIs, etc.) conditionaltests performed on data within one or more specified cells. Thesespreadsheet functions provide another form of data visualization basedon data in the spreadsheet. One aspect of the invention focuses onadapting and expanding this capability. This is represented in row I ofFIG. 1.

FIG. 2A illustrates a region 200 of a traditional spreadsheet or tableof tabular data. Such spreadsheet or tabular data can be created bydesktop applications such as Microsoft Excel or online applications suchas Google Spreadsheet. Aspects of the present invention can beimplemented via an add-on package of existing software or additionalplug-in script on any online applications. The minimum block that makesup this whole region is referred to as a cell, and cells can be mergedto compose a cell with bigger area expanding multiple rows, columns, orboth. Region 200 of an example spreadsheet contains four columns 281-284and five rows 291-295. Cell 230 is confined within one row 293 and oncolumn 283 and accordingly is defined by a left border 213, a rightborder 214, an upper border 222, and a lower border 223. Each of thefour columns 281-284 represents a different categorical subset of data(for example, an item, region, revenue area, inventory level, etc.). Incontrast to exemplary cell 230, another exemplary cell 240 in theleft-most column comprises the height of three rows 293, 294, 295.

The border of cell 230 is bolded with respect to the other cellsdepicted so as to call this cell out from the others. This can berendered for any of a number of reasons, such as signifying theselection of the cell via a pointing device of a user interface,selection resulting from a query, the signifying of special status orconditions, etc. The data displayed in cell 230 is a decimal numberformatted as a national financial currency and with athousands-separating comma. In contrast, the data displayed in cell 260is an unformatted non-decimal number without a thousands-separatingcomma. Also in contrast, the data displayed in cell 250 comprises a textstring, while the data displayed in cell 240 comprises an alphabetictext string and an unformatted non-decimal number. In the latter, theunformatted non-decimal number can in fact be part of the preceding textstring, or can be a calculated, retrieved, or can be a fixed numbercomposed with a calculated, retrieved, or fixed text string.

In general the data displayed in the foreground of a cell can containnumbers, text, symbols, or a combination of these, and this data orportions of it can be further “emphasized” with attributes. Suchattributes include choice of the font family, font size, bolding,italicizing, stretching, color, transparency with respect to the cellbackground, position within the cell, various types of animations(modulations of aforementioned attributes, blinking, shimmering,vibrating, rotating, etc.) Each individual font attribute can be fixedor determined by data values via conditional formatting. The datadisplayed in the foreground of a cell can be fixed, replicated, orresult from processing of other data by (no displayed) formulas and/orfunctions.

FIG. 2B depicts additional detail of cell 230 and its immediateneighborhood. In order to render the bolded border of the cell, thedividing border lines 213, 214, 222, 223 of FIG. 2A can be segmentedinto a plurality of sections—for example dividing border line 213 ofFIG. 2A can be segmented into at least adjacent sections 213 a, 213 b,213 c; similarly dividing border line 214 of FIG. 2A can be segmentedinto at least adjacent sections 214 a, 214 b, 214 c, dividing borderline 222 of FIG. 2A can be segmented into at least adjacent sections 222a, 222 b, 222 c, and dividing border line 223 of FIG. 2A can besegmented into at least adjacent sections 223 a, 223 b, 223 c. Theexemplary cell 230 is visually defined as the geometric area enclosed byupper, lower, left, and right dividing border lines 213 b, 214 b, 222 b,and 223 b—these border line sections can be accented via thickening,darkening, coloring, etc. so as to visually contrast with the othersections 213 a, 213 c, 214 b, 214 c, 222 a, 222 c, 223 a, 223 c. Suchborder line section accenting can be fixed or determined by data valuesvia conditional formatting.

The background 245 of each cell can be used to display a backgroundcolor, shading pattern, image, etc. which can be fixed or determined bydata values via conditional formatting. The background can vary fromcell to cell.

The displayed data in this particular case is, again, a decimal 241number formatted as a national financial currency via a signifyingcurrency symbol 242 and further employing a thousands-separating comma243.

FIG. 3 depicts of data and cell presentation attributes serving ascandidates for visualization parameters within a spreadsheet display.Some of these attributes can be controllable under conditionalformatting in traditional spreadsheets while others may not becontrollable under conditional formatting in traditional spreadsheets.In one aspect of the invention, a data or cell presentation attributeprovided a pre-defined range of variability and is controlled by aparameter that can vary over a pre-defined range (for example, between 0and 1, between 0 and 100, between −1 and +1, etc). Thisparameter-controlled variability of a data or cell presentationattribute allows that presentation attribute to be used as avisualization parameter.

For those data and cell presentation attributes that have beencontrollable under conditional formatting in traditional spreadsheets,the parameter-controlled variability provided for by the invention isfundamentally different in a number of ways. A few of these include:

-   -   Conditional formatting provides a particular custom-specified        result for satisfaction of a custom-specified rule;    -   For each individual result desired, a corresponding        hand-specified rule must be entered; for example in order to        span 64 color steps, 64 hand-specified rules must be        hand-entered;    -   In the case of colors, shades, fonts, stipples, etc.,        conditional formatting provided (or extended by VBA, APIs, etc.)        in traditional spreadsheets offers only a small discrete set of        choices. In the case of colors, Microsoft Excel for example        provides a fixed pre-selected collection of 56 colors, some of        which are duplicates. Even if Microsoft Excel provided 64        colors, this would only permit 4 steps per primary color in an        RGB color model or a very limited color wheel in a HSB color        model. Additionally, the fixed collection of pre-selected colors        will always have a significant portion of colors that will be        unusable as foreground or background colors as they will not be        able to visibly stand out, respectively, against a corresponding        range of background or foreground colors. Thus there are        typically considerably less than the full collection of        pre-selected colors to work with in a given color-control data        presentation visualization if numeric or text information is        co-displayed. Clearly the present invention provides a far        superior approach with greater capability, practicality, and        ease of use.

Among other capabilities, the invention provides new visualizationcapabilities to spreadsheet presentation, spreadsheet formatting,spreadsheet data handling, and spreadsheet interactive use in wayswell-suited for use as a numeric-intensive data visualization tool. Manyof these spreadsheet features can be implemented with simple vectorgraphics rendering operations or easily managed mixtures of vector andraster graphics, permitting:

-   -   Implementation of a number of the invention's spreadsheet        presentation features through use of APIs of existing        spreadsheet products;    -   Implementation of many of the invention's spreadsheet        presentation features with relatively small augmentation and        modification of existing product software by product        manufacturers;    -   Implementation of most of the invention's spreadsheet        presentation features with well-established graphics utilities        such as SVG (http://www.w3.org/TR/SVG11/) and the graphics        utilities of various computer operating systems provided by        companies such as Microsoft and Apple;    -   Via these and/or other graphics utilities, implementation of        most of the invention's spreadsheet presentation features in        browser-rendered web applications, thus making the invention's        spreadsheet presentation features available in a web page;    -   Via web and/or other implementation approaches, implementation        of the invention's presentation features in a collaborative        viewing environment.

The invention provides for a value of a data element, or result of aformula calculated from one or more data elements in a spreadsheet ordata table, to be used to determine spreadsheet presentation attributes,such as one or more of the following examples:

Attributes of a Font and/or Symbol:

-   -   the color of a font employed in rendering some data element        within the spreadsheet or data table;    -   the opacity of a font employed in rendering some data element        within the spreadsheet or data table;    -   the selection of a font employed in rendering some data element        within the spreadsheet or data table;    -   the embellishment of a font (bolding, italicizing, underlining,        etc.)    -   employed in rendering some data element within the spreadsheet        or data table;    -   the size of a font employed in rendering some data element        within the spreadsheet or data table;

Attributes of Appended Symbols:

-   -   the condition for the addition of an appending symbol or string        employed in rendering some data element within the spreadsheet        or data table;    -   where to append symbol or string    -   color(s) of symbol or string    -   opacity of symbol or string    -   other parameters of the symbol or contents of string;

Attributes of Numerical Quantization of Cell Data:

-   -   the number of decimal places employed in rendering some data        element within the spreadsheet or data table;    -   the method of numerical rounding employed in rendering some data        element within the spreadsheet or data table;    -   the degree of numerical rounding employed in rendering some data        element within the spreadsheet or data table;    -   the selection of a symbolic data element rendered within the        spreadsheet or data table;    -   the selection of an image element rendered within the        spreadsheet or data table;

Attributes of a Cell Border:

-   -   the color of at least one border of a cell enveloping a font        employed in rendering some data element within the spreadsheet        or data table;    -   the thickness/width of at least one border wall of a cell        enveloping a font employed in rendering some data element within        the spreadsheet or data table;    -   the thickness/width of at least one border outlining of a        plurality of contiguous cells, where each cell envelopes a font        employed in rendering of data elements within the spreadsheet or        data table;

Attributes of Cell Interior:

-   -   the background color of a cell enveloping some data element        within the spreadsheet or data table;    -   the background texture of a cell enveloping some data element        within the spreadsheet or data table;    -   the location of data element within a cell enveloping that data        element within the spreadsheet or data table;

Attributes of Rows & Columns:

-   -   an attribute of a column employed in rendering a plurality of        data elements within the spreadsheet or data table;    -   an attribute of a row employed in rendering a plurality of data        elements within the spreadsheet or data table.    -   an attribute of a row-by-column area employed in rendering a        plurality of data elements within the spreadsheet or data table.

The invention also provides for one or more of the followingcapabilities:

-   -   reordering of tabular rows and/or columns as a function of data        values;    -   superimposing a third (height) dimension atop the 2-dimensional        tabular data layout of an interactive electronic spreadsheet.    -   adopting the interactive format and metaphor of an interactive        electronic spreadsheet for use in a tabular presentation of        complex data visualization renderings;

Additionally, the invention provides for the control of data and cellpresentation attributes through use of a uniform parameterizationframework. This allows pre-visualization operations, such as scaling,translation, filtering, array (matrix, tensor) operations, nonlinearwarping, etc. to be employed in a modular, cascadable fashionindependent of the particular choice of data and cell presentationattributes. The invention further provides for pre-visualizationoperations to themselves have parameters that can be adjusted in realtime and/or be stored in files for recall. The invention furtherprovides for a network of pre-visualization operations to be stored infiles for recall.

The invention additionally provides for advanced user interface devices,particularly those providing large numbers of simultaneously-adjustableinteractive control parameters, to be used to control the viewing,presentation, and creation of the visualization as well as controllingthe underlying data source such as databases, statistical packages,simulations, etc.

Further, via web and other implementation approaches, the implementationof the invention's presentation features in a collaborative interactiveuse environment.

The invention thus can serve a variety of callings. FIGS. 4 a-4 eillustrate several example application settings where the invention iswell-suited to add value. In one scenario, depicted in FIG. 4A, aspreadsheet is initially used as a tool of choice or convenience 401,and at some point some simple data visualizations are incorporated 402.Such capabilities can be within the range provided by availablespreadsheet software products 400. As study of the data or developmentof the model intensifies, additional degrees of visualization areincorporated 403, potentially involving a need for a rich suite ofvisualization capabilities to create data visualization of substantialrichness 404. In FIG. 4B, a spreadsheet is chosen as a vehicle or venuefor visualization as it matches one dominating user needs (for example,original data already in or readily imported into a spreadsheet, quickassembly, tabular nature of data, tabular nature of formula replication,etc.). In FIG. 4C, a spreadsheet is chosen as a vehicle or venue forvisualization as it is comfortable or intuitive for a particularaudience. In FIG. 4D, a spreadsheet is chosen as a vehicle or venue forvisualization as other aspects of the invention allow for it to berendered in a web-based application. In FIG. 4E, a spreadsheet is chosenas a vehicle or venue for visualization as other aspects of theinvention allow for it to be rendered in a collaborative context.

The spreadsheet visualization aspects of the present invention are aspecial case of other more general aspects of the invention including:

-   -   the use of arbitrary or integrated data sources (such as static        databases, dynamic databases, streaming databases, live sensing        data streams, numerical simulations, signal processing,        statistical processing, linear and nonlinear transformations,        etc.);    -   uniform parameterizations of selected or all visualization        presentation parameters;    -   the support for real-time updates to integrated data sources        (such as static databases, dynamic databases, streaming        databases, live sensing data streams, numerical simulations,        signal processing, statistical processing, linear and nonlinear        transformations, etc.);    -   the use of data flow paths to link arbitrary data sources with        arbitrary data destinations via arbitrary topologies        (graphically, via an interconnection, specification, and/or        data-flow language, etc.);    -   the providing of shared GUI environments for controlling two or        more of visualization rendering, pre-visualization operations,        and data sources.

FIG. 5 illustrates an arrangement of these more general aspects of theinvention. Implicit in FIG. 5 are more general aspects of the invention,not restricted to spreadsheet visualization, that support visualrendering in a browser window and as a web application. As shown in thefigure, graphical user interfaces (GUIs) are provided for including oneor more relating to data sources and one or more relating tovisualization processes. Additionally, the invention also provides formultifunction GUIs that provide roles involving both data source mattersand visualization processes. The overall collection of user interfacesprovides the following capabilities:

-   -   Setup of a fixed data analysis configuration;    -   Setup of fixed data visualization configuration;    -   Setup of fixed data visualization presentation;    -   Setup of an interactive data analysis configuration;    -   Setup of an interactive data analysis session;    -   Setup of an interactive data visualization configuration;    -   Setup of an interactive visualization session;    -   Interactive control of data source selection and usage;    -   Interactive control of data analysis sessions;    -   Interactive control of visualization sessions;    -   Storage and recall of configurations;    -   Storage and recall of data;    -   Storage and recall of visualizations;    -   Storage and recall of sessions.

Yet additional aspects of the invention, not restricted to spreadsheetvisualization, also include (real-time and non-real-time) collaborationcapabilities. FIG. 6 illustrates an adaptation of the arrangement ofFIG. 5 to a collaboration environment supporting collaboration featuressuch as one or more of file exchange, real-time or stored displaysharing, real-time control sharing, real-time or stored collaborativeannotation, and archives of collaboration sessions and transactions.

Uniform Parameterizations of Cell and Data Presentation Attributes

The invention provides for uniform parameterizations of selected or allvisualization presentation parameters. This allows pre-visualizationoperations, such as scaling, translation, filtering, array (matrix,tensor) operations, nonlinear warping, etc. to be employed in a modular,cascadable fashion independent of the particular choice of data and cellpresentation attributes.

Data Flow Paths to Implement Arbitrary Interconnection Topologies

The invention provides for the use of data flow paths to link arbitrarydata sources with arbitrary data destinations via arbitrary topologies.This allows the selection and/or fusion of data sources, theirinterconnection with selected signal processing, statistical processing,pre-visualization operations, and visualization parameters (such as,among other examples, the cell and data presentation parameters of thespreadsheet visualization described earlier and in more detail tofollow).

FIG. 7 depicts an exemplary topological interconnection of data flowpaths linking various elements depicted in FIG. 5. In variousembodiments, functions such as data re-indexing, statistical processing,and signal processing can be provided as the data sources depicted inFIG. 5 or as the pre-visualization functions depicted in FIG. 5.Similarly, numerical simulations, for example (but not limited to) thoserendered by the computer running a spreadsheet program, ahigh-performance or other computer, can serve as the data sourcesdepicted in FIG. 5. Certain pre-visualization functions, for examplelinear predictors, can in an embodiment be regarded as a numericalsimulation. Additionally, in a spreadsheet visualization setting,calculations in the spreadsheet can provide a form of numericalsimulation prior to visualization.

The invention provides for some or all of the data flow paths (such asdepicted in the example of FIG. 7) to be specified in any convenientway, for example graphically via an interactive GUI or via acharacter-based language (interconnection, specification, and/ordata-flow, etc.). In an exemplary embodiment, a GUI can permit therendering of a graphic similar to that of FIG. 7. A GUI can permitcreation and customization of instances of functional blocks such as theones depicted in FIG. 7 from a library, menu, and/or graphical pallet.In an exemplary embodiment, a GUI can be used to create and link thesecustomized instances of functional blocks, via link-by-link “drawing,”with a data path topology such as the ones depicted in FIG. 7.

Various types of user interfaces may be used to create configurationssuch as that of FIG. 7 and set the various parameters within it. As oneexample approach, the selection of function blocks and the generalconnectivity among them can be specified using a drawing tool and apalette of function blocks, and clicking on each function block wouldcause dialog windows to appear that can be used for setting parameters.The parameters can be set by means of typed-in values, sliders, mousemanipulation, or the advanced high-dimension user interface devices tobe described later in the specification. More generally, a wide range ofapproaches, styles, and forms of configuration-specification,feature-selection, and parameter-setting GUIs are known in the art. Afew example user interfaces or aspects of them will be mentioned toshowcase different aspects, but a comprehensive treatment of userinterface approaches is not necessary for enablement as general forms ofconfiguration specification GUIs and parameter setting GUIs are known inthe art. A number of example configuration-specification,feature-selection, and parameter-setting GUIs will be the subject of alater companion patent application.

Pre-Visualization Operations

Attention is now directed to consideration of pre-visualizationoperations. FIGS. 8A-8C depict an approach for mapping a data valuelying within a pre-defined range to a value within a pre-defined rangefor a parameterized data or cell presentation attribute. In most casesthe input data range must be at least scaled and shifted to match thepre-defined range for a parameterized presentation attribute. This(linear or affine transformation) arrangement is depicted in FIG. 8A. Insome circumstances it can also be desirable to warp the data range withnonlinearity. In one embodiment a library of fixed or adjustablenonlinearities are provided which are such that the input and output ofthe nonlinearity both match the pre-defined range for a parameterizedpresentation attribute. This arrangement is depicted in FIG. 8B. Inanother embodiment the warping effect is provided with additionalflexibility by allowing pre-scaling and pre-shifting prior to applying aselected nonlinearity and subjecting the outcome of the nonlinearwarping to post-scaling and/or post-shifting operations in order tomatch the resulting range to the pre-defined range for a parameterizedpresentation attribute. This arrangement is depicted in FIG. 8C.Features and parameters of the above operations can be selected via userinterface dialog windows. General forms configuration-specification,feature-selection, and parameter-setting GUIs are known in the art.

FIG. 9 depicts a more general view and organization of pre-visualizationoperations provided for by the invention. In this example, availablepre-visualization operations include:

-   -   Data indexing/re-indexing, data sorting, data suppression, and        similar types of data operations;    -   Normalization, shifting (translation), and other types of linear        and affine transformations;    -   Linear filtering, convolution, linear prediction, and other        types of signal processing operations;    -   Warping, clipping, nonlinear transformations, nonlinear        prediction, and other nonlinear transformations.    -   The invention provides for other types of pre-visualization        operations as well.        Features and parameters of the above operations can be selected        via user interface dialog windows. General forms        configuration-specification, feature-selection, and        parameter-setting GUIs are known in the art.

The invention also provides for the inclusion of statistical operationsand statistical processing functions and for the linking to externalprograms to perform other types of pre-visualization operations.Features and parameters of these can be selected via user interfacedialog windows. General forms configuration-specification,feature-selection, and parameter-setting GUIs are known in the art.

The invention also provides for external programs to be added to thecollection of available pre-visualization operations.

The invention additionally provides for the inclusion and use of visualmetaphors to simplify visualization setup and user interaction for dataexploration. As an example, FIG. 10 depicts an exemplary embodimentwherein a selected metaphor is used to automatically generate parameterassignments and graphics rendering operations. The invention providesfor metaphors to control other aspects of the visualization andpre-visualization operations. The invention provides for a metaphor tobase its operations on characteristics of a data set being visualized,previously visualized, and anticipated to be visualized. The inventionadditionally provides for metaphors to be selected and controlled byuser interaction, data values, or other means. Features and parametersof the above operations can be selected via user interface dialogwindows. General forms configuration-specification, feature-selection,and parameter-setting GUIs are known in the art.

The invention also provides for array (vector, matrix, tensor)operations such as (vector, matrix, tensor) linear combinations,(vector, matrix, tensor) multiplication, scalar multiplication, finding(matrix, tensor) determinants, finding (matrix, tensor) inverses andpsuedoinverses, row reduction, factorization, change of basis, orcalculation of an eigensystem (eigenvalues, eigenvectors, eigentensors).

Users can select the region of the spreadsheet to be stored in an arrayand give a name to the array. Different programs can require differentformatting of arrays, thus, users can also choose the separatingcharacters, such as a comma and a space or tab. The key of the array canbe numbers from 0 to (the number of entries—1) or the values of anycolumn user selects. Such array created can be exported to an externalprogram. Similarly, data of one or multiple arrays can be implemented tobe imported from an external program and the values of elements aredisplayed in appropriate numbers of rows and columns. FIG. 11Aillustrates an example of a user interface that permits users to selectthe geometric element boundary of a captured or rendered tabular matrixdata elements or cells within a spreadsheet. Other features andparameters of the above operations can be selected via user interfacedialog windows. General forms configuration-specification,feature-selection, and parameter-setting GUIs are known in the art.

In the case of spreadsheet visualization, spreadsheets natively providecertain types of array calculations with relative ease while otherstandard matrix and array operations, such as matrix/tensormultiplication, eigensystem calculations, convolutions, etc., have nonative support within the spreadsheet paradigm. The invention providesfor a matrix boundary tool for use in capturing and labeling matrix dataand rendering matrix operation results. For example, users can specifythe group of cells as the elements of one matrix or multiple matrices.FIG. 11B illustrates an exemplary user interface that permits users toselect array operations to be performed on selected array data, forexample as may be selected by the tool of FIG. 11A. Other features andparameters of the above operations can be selected via user interfacedialog windows. General forms configuration-specification,feature-selection, and parameter-setting GUIs are known in the art.

The invention can additionally transform a graphics object via matrixoperations. As examples, FIG. 12A illustrates a figure being resized,rotated, and shifted through array-based scaling, rotating, andtranslating linear transformations. FIG. 12B illustrates processes of afigure being rotated around x-axis, y-axis, line of y=x, and line ofy=−x. and FIG. 12 c illustrates processes of figure being rescaledvertically or horizontally. FIG. 12D illustrates a transformationprocess acting on 2D graphics objects such as a rectangle-boundedhorizontal line array and a graphic. The invention also provides forembodiments to extend these and related systems and methods to transform3D-graphics via scalings, rotations, stretch, etc.

The invention provides for dimensional transformations among points, 1D,2D, and 3D graphics objects. FIG. 12E illustrates a matrix and/or tensortransformation process and how it changes a 3D graphics object into a 2Dobject via dimension dropping. FIG. 12F illustrates a 3D graphics objectrendered of a 2D surface created by a vertical line rotated around they-axis. FIG. 12G illustrates a 3D graphics object rendered byperspective projection defined by a point.

The invention also provides for 3D graphics generation from 3D vectordraw-lists provided by other programs, spreadsheet data, and/or theuser. In an embodiment, the invention also provides for 3D graphicsgeneration from equations provided by the user.

The invention further provides for pre-visualization operations tothemselves have parameters that can be adjusted in real time and/or bestored in files for recall.

The invention further provides for a network of pre-visualizationoperations to be stored in files for recall.

Spreadsheet Visualization Integration Architectures

The invention provides implementation of at least some of theinvention's spreadsheet presentation features through use of APIs ofexisting spreadsheet products. The invention provides implementation ofat least some of the invention's spreadsheet presentation features withrelatively small augmentation and modification of existing productsoftware by product manufacturers.

FIG. 13 depicts an example of spreadsheet architecture (right side) andan information flow as it might be used in a data visualization setting.One of or some combination of entered data, measured data, and/orcomputed data can be stored as stored data or stored in a queriable database. In some situations live data feeds can also provide data. One orsome combination of these data sources can be used to provide a data setused by the spreadsheet program. The spreadsheet program internallycomprises at least a GUI component, a calculation compound, and a visualrendering component.

FIGS. 14A-14D depicts various types of exemplary relationships andinformation flows among a system or method for rendering a conventionalspreadsheet or table of tabular data, a system or method for renderingthe visualization operations of the invention, and a data set used byeach. Visualization and pre-visualization operations can be introducedbetween the data and the spreadsheet program as depicted in FIG. 14A. Inanother embodiment, visualization and pre-visualization operations canobtain data from the spreadsheet program, process data, and return thegenerated visual operation back to the spreadsheet program as depictedin FIG. 14B. An example of this is using an API or the use of VBA inconjunction with Microsoft Excel.

Visualization and pre-visualization operations can also be implementedas parallel operations to the spreadsheet program. Visualization andpre-visualization operations can be configured separately as anindependent operation(s) with direct access to data (i.e., able toobtain data and send the result to the data without having to go throughthe spreadsheet program), as shown in FIG. 11C. One advantage of this isthat relevant categories of data can be extracted to generatecomplicated visualizations without the data being arranged into tableform. Visualization and pre-visualization operations can be part of aspreadsheet program without direct access to the data, as depicted inFIG. 14D.

Spreadsheet Visualization User Interface Architectures: Traditional andHigh-Dimension User Interface Devices (HDTP and Advanced Mice)

Next, user interface architectures for general visualization andspreadsheet visualization environments are considered. FIG. 15 depictsan exemplary embodiment of an augmentation of a spreadsheet program withadditional spreadsheet features as provided for by the invention. A dataset is subjected to data operations so as to produce modified data. Thedata operations can be controlled by the spreadsheet GUI and spreadsheetcalculation as well as by the visualization GUI and Visualizationcalculations.

FIGS. 16A-16B depict ways in which one or more user interface inputdevices (such as a mouse, trackball, touchpad, etc.) can be managed tocontrol graphical user interfaces of a plural software application inthe context of the invention. FIG. 16A, the windowing system directs theinput events from a single user interface input device to variousapplications, for example, a Database GUI, a Simulation GUI, aSpreadsheet GUI, and a Visualization GUI. FIG. 16B depicts analternative implementation wherein a Visualization Environment InputRouter/Parser accepts input events from one or more user interface inputdevices and directs these events to each of the various applications.The Visualization Environment Input Router/Parser can direct events frommultiple user interface input devices to different inputs to the sameapplication. In another embodiment the Visualization Environment InputRouter/Parser can split events originating from an individual userinterface device and separately route these to inputs of separateapplications. In another embodiment, a Visualization Environment InputRouter/Parser works together with the Window System to route inputevents.

The above arrangements are general and apply to use of conventionalmice, trackballs, touchpads, etc. However, visualization and CADworkstations have often been provided with more sophisticated user inputdevices that provide a higher number of interactivesimultaneously-adjustable parameters. Classic examples of this areknob-boxes (as used in HP and SGI workstations), the DataGlove (VPL,General Reality), the SpaceBall (Logitech3Dconnexion, Labtec,HP/Compaq), etc., although few of these have survived product cycles toremain in active use or with wide availability. More recently enhancedtouch-based interfaces have attracted a great deal of attention, mostlyfor their multi-touch and gesture recognition capabilities. However,some enhanced touch-based interfaces such as the HDTP (“High DimensionalTouch Pad,” U.S. Pat. No. 6,570,078; U.S. patent application Ser. Nos.11/761,978 U.S. Ser. No. 12/418,605, and U.S. Ser. No. 12/502,203 amongothers) employ a tactile sensor array (pressure, proximity, etc.) andreal-time image and mathematical processing to provide a powerful userinput device with both a higher number of interactivesimultaneously-adjustable parameters and a rich range of syntactic andmetaphorical capabilities well-suited to use with interactivevisualization. Additionally, the HDTP technology can be readilyimplemented as a touchscreen through use of, for example, inexpensivetransparent capacitive proximity-sensor arrays. The present inventionprovides for the incorporation and use of the HDTP and othermetaphor-rich user interface input devices into visualizationenvironments. Attention is now directed to incorporation of the HDTP aspart of a visualization environment user interface and then briefattention is directed to alternative use of a few alternativemetaphor-rich enhanced user interface technology approaches.

FIGS. 17A-17F illustrate the six independently adjustable degrees offreedom of touch from a single finger that can be simultaneouslymeasured by the HDTP technology. The depiction in these figures is fromthe side of the touchpad. FIGS. 17A-17C show actions of positionalchange (amounting to applied pressure in the case of FIG. 17C) whileFIGS. 17D-17F show actions of angular change. Each of these can be usedto control a user interface parameter, allowing the touch of a singlefingertip to control up to six simultaneously-adjustable quantities inan interactive user interface. FIG. 18 suggests general ways in whichtwo or more of these six independently adjustable degrees of freedom1811-1816 can be adjusted at once for a finger 1800 in contact with anHDTP sensor surface 1801. More advanced implementations of the HTDPprovide for multi-touch capabilities far more sophisticated than thosepopularized by the Apple iPhone, NYU, and others.

FIG. 19 demonstrates a few exemplary two-finger multi-touch posturesand/or gestures from the hundreds that can be readily recognized by HTDPtechnology. The two fingers may be spread apart somewhat, as in thecases 1901-1903 and 1931-1933, or may be brought together, as in thecases 1911-1913 and 1941-1943. The separation of the two fingers adds anadditional parameter that can be controlled by the user, and the yaw(pivot) parameter may be varied independently from the newly introducedfinger separation parameter. For example the hand can pivot clockwise1902, 1912, or counter-clockwise 1903, 1913 from the correspondingneutral positions 1901, 1911. Similarly, the pitch of the pair offingers can be raised 1932, 1942 or lowered 1933, 1943 with respect tothe corresponding neutral positions 1931, 1941, while independently fromthese motions the two fingers are spread apart 1931-1933 or broughttogether 1941-1943. Note the finger spread can be varied considerably inthis two-finger posture.

The pair of fingers can be moved as a group through all six degrees offreedom (left-right, forward-back, downward pressure, roll, pitch, yaw),and comfortably allow for two differences between the fingers(differences in downward pressure, and one angle of finger separationspread) and even three differences between the fingers (differences indownward pressure and two coordinates of separation when fingers curl toform independently controlled “x” and “y” components). Thus two-fingerpostures considered above can readily provide a nine-parameter setrelating to the pair of fingers as a separate composite objectadjustable within an ergonomically comfortable range. One examplenine-parameter set the two-finger postures comprises:

-   -   composite (group) average x position;    -   inter-finger differential x position;    -   composite (group) average y position;    -   inter-finger differential y position;    -   composite (group) average pressure;    -   inter-finger differential pressure;    -   composite (group) roll;    -   composite (group) pitch;    -   composite (group) yaw.

HTDP technology can also be configured to recognize and measure posturesand/or gestures involving three or more fingers, various parts of thehand, the entire hand, multiple hands, etc. In general, multifingercontact can be used to provide control of up to three additionalindependently adjustable parameters for each additional finger.

FIG. 20 shows an example of how raw measurements of the six quantitiesof FIGS. 17A-17F, together with shape recognition for distinguishingcontact with various parts of the hand and the touchpad, can be used tocreate a rich information flux of parameters, rates, and symbols. FIG.21 shows an exemplary approach for incorporating posture recognition,gesture recognition, state machines, and parsers to create an evenricher human/machine tactile interface system capable of incorporatingsyntax and grammars.

FIG. 22 depicts an exemplary user interface input arrangementincorporating one or more HDTPs that provides a user interface inputevent and quantity routing of the type described earlier in conjunctionwith FIGS. 16 a-16 b.

FIGS. 23A-23L depict a number of arrangements employing the HDTPtechnology suitable for use with visualization environments. FIG. 23Aillustrates an HDTP as a peripheral that can be used with a desktopcomputer (shown) or laptop (not shown). FIG. 23B depicts an HDTPintegrated into a laptop in place of the traditional touchpad pointingdevice. In FIGS. 23A-23B the HDTP tactile sensor can be a stand-alonecomponent or can be integrated over a display to form a touchscreen.FIG. 23C depicts an HDTP integrated into a desktop computer display toform a touchscreen. FIG. 23D shows the HDTP integrated into a laptopcomputer display to form a touchscreen. FIG. 23E depicts an HDTPintegrated into a cellphone, smartphone, PDA, or other hand-heldconsumer device. FIG. 23F shows an HDTP integrated into a testinstrument, portable service-tracking device, portable service-entrydevice, field instrument, or other hand-held industrial device. In FIGS.23E-23F the HDTP tactile sensor can be a stand-alone component or can beintegrated over a display to form a touchscreen. FIG. 23G depicts anHDTP touchscreen configuration that can be used in a tablet computer,wall-mount computer monitor, digital television, video conferencingscreen, kiosk, etc. In at least the arrangements of FIGS. 23A, 23C, 23D,and 23G, or other sufficiently large tactile sensor implementation ofthe HDTP, more than one hand can be used and individually recognized assuch.

FIGS. 23H-23K depict various integrations of an HDTP into the back of aconventional computer mouse. In FIGS. 23H-23K the HDTP tactile sensorcan be a stand-alone component or can be integrated over a display toform a touchscreen. More than two touchpads can be included, such as theexample of FIG. 23L as taught in U.S. Pat. No. 7,557,797.

The types of human-machine geometric interaction between the hand andthe HDTP facilitate many useful applications within a visualizationenvironment. A few of these include control of visualization observationviewpoint location, orientation of the visualization, and controllingfixed or selectable ensembles of one or more of viewing parameters,visualization rendering parameters, pre-visualization operationsparameters, data selection parameters, simulation control parameters,etc. As one example, the 6D orientation of a finger can be naturallyassociated with visualization observation viewpoint location andorientation, location and orientation of the visualization graphics,etc. As another example, the 6D orientation of a finger can be naturallyassociated with a vector field orientation for introducing syntheticmeasurements in a numerical simulation. As another example, at leastsome aspects of the 6D orientation of a finger can be naturallyassociated with the orientation of a robotically positioned sensorproviding actual measurement data. As another example, the 6Dorientation of a finger can be naturally associated with an objectlocation and orientation in a numerical simulation. As another example,the large number of interactive parameters can be abstractly associatedwith viewing parameters, visualization rendering parameters,pre-visualization operations parameters, data selection parameters,numeric simulation control parameters, etc.

In another example, the x and y parameters provided by the HDTP can beused for focus selection and the remaining parameters can be used tocontrol parameters within a selected GUI.

In another example, the x and y parameters provided by the HDTP can beregarded as specifying a position within an underlying base plane andthe roll and pitch angles can be regarded as specifying a positionwithin a superimposed parallel plane. In a first exemplary extension ofthe previous two-plane example, the yaw angle can be regarded as therotational angle between the base and superimposed planes. In a secondexemplary extension of the previous two-plane example, the fingerpressure can be employed to determine the distance between the base andsuperimposed planes. In a variation of the previous two-plane example,the base and superimposed planes are not fixed as being parallel butrather intersect at an angle associated with the yaw angle of thefinger. In each of these examples, either or both of the two planes canbe used to represent an index or indexed data, a position, pair ofparameters, etc. of a viewing aspect, visualization rendering aspect,pre-visualization operations, data selection, numeric simulationcontrol, etc.

A large number of other examples are possible as is appreciated by oneskilled in the art.

One use of the HDTP in the above examples is simply to supply more thanthe usual two user interface parameters provided by a conventional userinterface input device such as a conventional computer mouse, trackball,touchpad, etc. The present invention provides for the use of other userinterface input arrangements and devices as alternatives to or inconjunction with one or more HDTPs.

In a simple example, the scroll-wheel of a scroll-wheel mouse is used toprovide a third simultaneously adjustable user interface parameter. Inanother example, a second or yet more additional scroll-wheels can beadded to a conventional scroll-wheel mouse. The resultant collection ofscroll-wheels can be relatively positioned in parallel, oriented atorthogonal angles to support a coordinate-metaphor, positioned on thesides of the mouse body, etc. FIGS. 24A and 24B illustrate examples2410, 2420 of a conventional scroll-wheel mouse 2401, 2402 with atraditional up-down scrollwheel 2421 and at least standard mouse buttons2411, 2412 provided with an added left-right scroll-wheel 2422 as taughtin U.S. Pat. No. 7,557,797. Such an arrangement can employ a connectingcable, or the device can be wireless. Each of the arrangements of FIGS.24A and 24B as well as the ones described in the next paragraph, plusvariations and adaptations, will be referred to as an “advanced mouse”or in the plural “advanced mice.”

In another example of an advanced mouse, one or more trackballs can beadded to a conventional computer mouse. FIGS. 25A-25C illustrateexamples 2510, 2520, 2530 wherein a single trackball 2511, 2521, 2531 isincorporated into the back of a conventional computer mouse 2510, 2520,2530 with at least standard mouse buttons 2512-2513, 2522-2523,2532-2533 as taught in U.S. Pat. No. 7,557,797. FIGS. 26A-26C illustrateexamples where two trackballs 2605 a, 2605 b are incorporated into theback of a conventional computer mouse as taught in U.S. Pat. No.7,557,797. The trackballs in the arrangements of FIGS. 25A-25C and FIGS.26A-26C can be the conventional two degree of freedom type (rollleft-right, roll away-towards) or can provide three to six degrees offreedom as taught in U.S. Pat. No. 7,557,797; U.S. Patent ApplicationUS-10/806,694. Useful exemplary advanced mouse arrangements include thetrackball 2605 a-2605 b, touchpad 2605 c, scrollwheel 2668 mouse of FIG.26C, the trackball/touchpad mouse of FIG. 26 d, and the multiple sliderconfiguration of FIG. 26E, each as taught in U.S. Pat. No. 7,557,797.Each of these arrangements can employ a connecting cable, or theadvanced mouse device can be wireless.

The additional parameters provided by the HDTP and the abovealternatives are more than the usual number supported by conventionalwindow systems (for example as described in conjunction with FIG. 16A).To implement additional parameter handling, for example such as in theHDTP arrangement of FIG. 22 or its equivalents for the alternativeadvanced mice as described above, additional arrangements must be madebeyond the conventional operating system pointer device handling.

In an additional approach, the invention provides for the HDTP oralternatives such as the advanced mice described above to interface witha browser via a browser plug-in. This arrangement can be used to capturethe additional user interface input parameters and pass these on to anapplication interfacing to the browser. An example of such anarrangement is depicted in FIG. 27. The browser can interface with localor web-based applications that drive the visualization and/or controlthe data source(s), process the data, etc. The browser can be providedwith client-side software such as JAVA Script, AJAX, etc. Such anarrangement is particularly useful when combined with browser-basedrendering of visualizations, as described in the next section.

The invention provides for HDTP parameters to be separated into groupswhich are individually directed to pointer device interfaces on multiplecomputers.

Browser-Rendered Implementations

The invention provides for visualizations to be rendered in a browser.This allows for implementations wherein the browser is used as a viewer.The browser can interface with local or web-based applications thatdrive the visualization. An exemplary arrangement is depicted in FIG.28. In an embodiment, the browser can be provided with Simple VectorGraphics (“SVG”) utilities (natively or via an SVG plug-in) to renderbasic 2D vector and raster graphics. In another embodiment, the browsercan be provided with a 3D graphics capability, for example via theCortona 3D browser plug-in. These embodiments or alternatives can beprovided with client-side software such as JAVA Script, AJAX, etc. Theexample of FIG. 28 also provides for other plug-ins (represented by thedashed-line box) such as the user interface input device plug-indescribed earlier.

Web-Based Implementations

The invention provides for web-based implementations of thevisualization environment. FIG. 29 depicts an exemplary web-basedimplementation of a visualization environment leveraging browserarrangements such as that depicted in FIGS. 27-28, their variations,alternatives, etc. As depicted in the arrangement of FIG. 29, the clientenvironment can employ such a browser to access a visualization serverenvironment or visualization-related (databases, live data feeds,simulations, statistical or signal processing, etc.) server environment.In an embodiment, the client environment browser can connect over theinternet with two or more visualization server environments and/orvisualization-related server environments and/or local applications. Inan embodiment, a visualization or visualization-related serverenvironment can connect over the internet with two or more browsers inthe client environment. Other variations and alternative implementationsare also possible as is clear to one skilled in the art.

A server environment can include web server foundation software such asApache, 115, etc. A server environment can also include server-sidescripting and dynamic software such as CGI, JAVA/JAVA Script, Python,PHP, Perl, JSP, ASP, etc. Other variations and alternativeimplementations are also possible as is clear to one skilled in the art.

The additional interactively-controlled parameters provided by the HDTPprovide more than the usual number supported by conventional browsersystems and browser networking environments. This may be addressed in anumber of ways.

In a first approach, an HDTP interfaces with a browser both in atraditional way and additionally via a browser plug-in. Such anarrangement may be used to capture the additional user interface inputparameters and pass these on to an application interfacing to thebrowser. An example of such an arrangement is depicted in FIG. 30D.

In a second approach, an HDTP interfaces with a browser in a traditionalway and directs additional GUI parameters though other network channels.Such an arrangement may be used to capture the additional user interfaceinput parameters and pass these on to an application interfacing to thebrowser. An example of such an arrangement is depicted in FIG. 30E.

In a third approach, an HDTP interfaces all parameters to the browserdirectly. Such an arrangement may be used to capture the additional userinterface input parameters and pass these on to an applicationinterfacing to the browser. An example of such an arrangement isdepicted in FIG. 30F.

The browser may interface with local or web-based applications thatdrive the visualization and/or control the data source(s), process thedata, etc. The browser may be provided with client-side software such asJAVA Script, AJAX, etc. The browser may provide also be configuredadvanced graphics to be rendered within the browser display environment,allowing the browser to be used as a viewer for data visualizations,advanced animations, etc., leveraging the additional multiple parametercapabilities of the HDTP. The browser may interface with local orweb-based applications that drive the advanced graphics. In anembodiment, the browser may be provided with Simple Vector Graphics(“SVG”) utilities (natively or via an SVG plug-in) so as to render basic2D vector and raster graphics. In another embodiment, the browser may beprovided with a 3D graphics capability, for example via the Cortona 3Dbrowser plug-in.

Collaboration Implementations

The invention provides for collaboration implementations and the use ofcollaboration tools for collaborative use of visualization features ofthe invention and for the creation of a collaborative visualizationenvironment.

FIG. 30A shows a client-side collaboration implementation wherein ageneral purpose collaboration tool can be used to share thevisualization environment session running on one computer with one ormore other computers. The visualization environment can be a stand-alonevisualization application or can comprise a browser implementationemploying, for example, any of the various arrangements described inconjunction with FIGS. 27-29.

In the case of real-time collaboration, the general purposecollaboration tool employed in the arrangement of FIG. 30 can compriseapplication sharing software (such as Microsoft's NetMeeting™,IBM/Lotus' SameTime™^(,) ^(SM), HP's Shared-X, etc.) or display sharingsoftware. In the case of real-time collaboration, audio and/or videoconferencing can also be used to facilitate interpersonal communicationamong collaborating session participants.

FIG. 30B shows server-side collaboration implementation wherein ageneral purpose or specialized server-side application sharingenvironment is used to share a server-based interactive datavisualization application.

FIG. 30C shows a server-side collaboration implementation comprising aserver-based multi-user interactive data visualization application.

Interfacing with Sonification

The invention further provides for interfacing with established andadvanced data sonification utilities. Data sonification has receivedconsiderable attention and analysis but for the most part is fairlysimplistic and often contributes little practical value. More advanceddata sonification techniques, such as the multi-channel datasonification system described in U.S. Patent Application 61/268,856employed in the environmental GIS (Geographic Information System)process monitoring and modeling system of U.S. Patent Application61/268,873, provides new opportunities for practical use of datasonification.

FIG. 31 illustrates a general framework for data sonification where aparameterized metaphor is used to manage parameter assignment and soundrendering and a parameterized data indexing operation.

Data sonification may be applied to the same data used to generatevisualization. The data directed to data sonification may be selected byinteracting with a rendered visualization via a user interface inputdevice. Data sonification may be provided by a multichannel datasonification system such as the one described in U.S. Patent Application61/268,856. Data sonification output may be shared using an audiochannel of a real-time collaboration system, may be transmitted from aweb-based application using an audio channel as provided by ahigh-fidelity VoIP system, may be shared using an audio channel asprovided by a high-fidelity VoIP system, may be produced local to theuser computer under the control of a web-based application, or may beproduced local to the user computer under the control of a web-basedapplication employing MIDI protocol.

Some user interface considerations relating to configurations, featuresand parameters of data sonification operations are described in theaforementioned U.S. Patent Application 61/268,856. Beyond these, andmore generally, configurations, features and parameters of datasonification operations can be selected via user interface dialogwindows. General forms configuration-specification, feature-selection,and parameter-setting GUIs are known in the art.

Uniform Parameterizations of Cell and Data Presentation Attributes

The invention provides for visualization rendering parameters to beuniform over a common variational range. This permits uniform handlingof visualization rendering parameters and numerical operationcompositions. Any of these can be set, varied, or modulated as selectedor as advantageous to represent data or information derived from data.In an embodiment, the invention provides for visualization renderinguniform parameters in the range of [0,1]. The uniform parameter range of[0,1] is used in the examples below, but it is understood that otherchoices for the uniform range are also possible and are provided for bythe invention.

Color Morphing

The invention provides for at least one color selection option to bedetermined by a uniform parameter in the range of [0,1]. FIG. 32 depictsan exemplary embodiment wherein the hue of a color can be varied throughnatural colors of red through violet for the bulk of the [0,1] range anda range of artificial colors, such as the purple through magentacontinuum, are appended for the remaining portion of the [0,1] range.The invention also provides for other color models, such as RGB, HSB,LUV, LAB, grayscale, etc. to be used with one to three uniformparameters, each in the range of [0,1]. Such uniform parameter colorscan be assigned one or more of text, symbols, borders, backgrounds,gridlines, glyphs, lines, components of lines, components of stipples,components of gradients, surfaces, and other geometric primitivesemployed in the visualization of data. Other variations and alternativearrangements are possible as is clear to one skilled in the art.

A uniform parameter range of [0,1] is used in the examples above, but itis understood that other choices for the uniform range are also possibleand are provided for by the invention.

Line Morphing

The invention provides for at least one line rendering option to bedetermined by a uniform parameter in the range of [0,1]. FIG. 33illustrates an example wherein line width is determined by a uniformparameter in the range of [0,1]. FIG. 33 also illustrates an examplewherein line dashing period is determined by a uniform parameter in therange of [0,1]. FIG. 33 further illustrates an example wherein linedashing duty-cycle is determined by a uniform parameter in the range of[0,1]. FIG. 33 also illustrates an example wherein line dashingsub-duty-cycle is determined by a uniform parameter in the range of[0,1]. Other variations and alternative arrangements are possible as isclear to one skilled in the art.

The invention provides for line color to be determined by a uniformparameter color model as described above and for line dashing to includea plurality of colors.

Stipple Morphing

The invention provides for at least stipple pattern rendering option tobe determined by a uniform parameter in the range of [0,1]. FIG. 34illustrates an example wherein stipple line width is determined by auniform parameter in the range of [0,1]. FIG. 34 also illustrates anexample wherein stipple gap is determined by a uniform parameter in therange of [0,1]. A period and duty cycle model can be used instead andcontrolled via respectively associated uniform parameters varying overthe range of [0,1].

FIG. 34 further illustrates an example wherein stipple angle isdetermined by a uniform parameter in the range of [0,1]. Othervariations and alternative arrangements are possible as is clear to oneskilled in the art.

The invention provides for at least one of stipple foreground andbackground colors to be determined by a uniform parameter color model asdescribed above. The invention provides for a stipple pattern tocomprise more than two colors, for the inclusion of line dashing in astipple pattern, and for line dashing to include a plurality of colors.

Other variations and alternative arrangements are possible as is clearto one skilled in the art. A uniform parameter range of [0,1] is used inthe examples above, but it is understood that other choices for theuniform range are also possible and are provided for by the invention.

Border Morphing

The invention provides for at least one border rendering option to bedetermined by a uniform parameter in the range of [0,1]. The borderrendering option can comprise the border of a spreadsheet cell asdepicted in FIG. 35, or can comprise the border of another visuallyrendered graphical object.

In the complete cell depicted in the upper right corner of FIG. 35, allfour sides of the cell border are uniform. Such a border to have acommon width W₀ controlled by a uniform parameter in the range of [0,1]and to have a common color controlled by one or more uniform parametersin the range of [0,1]. The cell border can be rendered as a gradientamong two or more selected colors, wherein the colors and/or gradientcharacteristic can be respectfully determined by an associated uniformparameter in the range of [0,1].

In the complete cell depicted in the lower right corner of FIG. 35, allfour sides of the cell border are distinct. The invention provides foreach such border segment to have an associated width (W₁-W₄), eachrespectively controlled by an associated uniform parameter in the rangeof [0,1]. The invention further provides for each such border segment tohave a common color controlled by one or more uniform parameters in therange of [0,1]. A border segment can be rendered as a gradient among twoor more selected colors, wherein the colors and/or gradientcharacteristic can be respectfully determined by an associated uniformparameter in the range of [0,1].

The invention further provides for such a cell (interior) background tobe filled with a color controlled by one or more uniform parameters inthe range of [0,1]. In an embodiment, the invention provides for stipplepatterns to be used as a cell background. The cell background can berendered as a gradient among two or more selected colors, wherein thecolors and/or gradient characteristic can be respectfully determined byan associated uniform parameter in the range of [0,1].

The invention also provides for the four sides of a cell border to besegmented in other ways, for example as in the two complete cells in theleft side of FIG. 35.

Other variations and alternative arrangements are possible as is clearto one skilled in the art. A uniform parameter range of [0,1] is used inthe examples above, but it is understood that other choices for theuniform range are also possible and are provided for by the invention.

Font Morphing

The invention provides for font morphing and font interpolation underthe control of uniform parameters in the range of [0,1]. Other choicesfor the uniform range are also possible and are provided for by theinvention.

FIG. 36A depicts a boldness parameter under the control of uniformparameters in the range of [0,1]. In an embodiment, the variation inboldness can be implemented so that it is governed by the opticalproperties of de Groot Interpolation Theory. In an embodiment, a minimumand maximum degree of boldness are assigned to the extremes of theparameter range.

FIG. 36B depicts an italics angle parameter under the control of uniformparameters in the range of [0,1]. In an embodiment, a minimum andmaximum degree of italics angle is assigned to extremes of the parameterrange.

FIG. 36C depicts an font family interpolation parameter under thecontrol of uniform parameters in the range of [0,1]. One font family isassigned to the minimum extreme of the parameter range and a second fontfamily is assigned to the maximum extreme of the parameter range.

FIG. 37A depicts vertical and horizontal dilation/contractionparameters, each respectively under the control of uniform parameters inthe range of [0,1]. In an embodiment, a minimum and maximum degree ofitalics angle are assigned to the extremes of the parameter range.

FIG. 37B shows font underline line width and font underline separationgap parameters, each respectively under the control of uniformparameters in the range of [0,1]. In an embodiment, a minimum andmaximum degree of italics angle are assigned to the extremes of theparameter range.

Other variations and alternative arrangements are possible as is clearto one skilled in the art. A uniform parameter range of [0,1] is used inthe examples above, but it is understood that other choices for theuniform range are also possible and are provided for by the invention.

User Interfaces for Adjusting Cell and Data Presentation Attributes

Configurations, features and parameters for related operations, such asdata set selection, data ranges, offsets, scalings, warping functions,etc., can be, for example, selected via traditional types of userinterface dialog windows. General forms configuration-specification,feature-selection, and parameter-setting GUIs are known in the art.

The invention provides for user interfaces to simplify operation with afriendly user interface for new or occasional users. A typical attributecontrol panel, as shown in FIG. 38, contains a few pre-selected valuesand a value which can be customized usually by selecting a value from adrop-down list.

Also, users can specify if the changes in attributes are to be appliedglobally or are to be applied to a single cell or a group of cells in aspreadsheet. The change to be made can also be determined by the valueof a data element or by the result of a formula calculated from one ormore data elements.

FIG. 39 illustrates an example of how change in color of a font can bespecified for a cell, a group of cells, or globally across an entireworksheet. Users can select the color of a font simply by clicking on aregion in the gray level or Hue color chart or by moving the slider thatdetermines the gray level or parameters in Hue. A formula thatdetermines such value can also be included. For example, the ratio ofthe data in a cell to the range of the data in a category can determinethe value of gray level accordingly.

FIG. 40 depicts an interface for determining a font type of thespecified cell(s). Such interface can have a pre-selected list or a listto be customized or a combination of both, and can be implemented in awide variety of ways.

FIG. 41 depicts an example of an interface for determining theembellishment of a font in rendering some data element within thespreadsheet or data table. The data element can be displayed in boldface, italics or underlined. The percentage of bolding or the thicknessof the underline can be varied. For example a data element can beemphasized by being displayed in bold face, or a title of a book orquotation can be displayed in italics, or a link to a website or areference to another cell or region can be underlined.

FIG. 42 depicts an example interface to control the size of a font to bedisplayed. Several preset sizes are listed on the panel as well as aslider that enables users to control the font size gradually. Font sizecan be incremental with step sizes or continuous, which can be adjustedby controlling a slider, as long as the value is positive and less thanreasonable size compared to the area of the text displayed.

The invention provides for a user interface to determine addition of anappending symbol. The symbol(s) can be different types, numbers, orsizes of graphical objects. The factor that determines can depend on thesignificance or the ratio of the element to the category or the value ofanother category in the dataset. In this example, the symbols appendedto the right end of the chart represent different categories ofexpenses. If the symbol appended is configured to represent quantity,the number of symbols to be appended will depend on the ratio of thevalue of the cell to the unit or base value. If the symbol appended isconfigured to represent significance, the size of symbols to be appendedwill depend on the significance of the value of the cell in thatcategory. FIG. 43 depicts an example wherein data in a cells in aselected column can have symbol appended to numbers or other data withinthe cells of the column.

Different formatting options can be applied to different types of data.The factor that determines the type of data can be obtained from theresult of a hidden formula, from associated data such as the data fromanother column in the same row, or simply selected by the user. Thenumber of decimals can be determined according to such factors. Alsosymbols can be attached according to the type of data. If the data in acell represents currency, a currency symbol can be attached, and if thedata represents percentage, a percentage sign will be attached. Furtherif the country is specified in another column, the correspondingcountry's currency sign can be attached.

The invention provides for the varying the color andopacity/transparency of the background color of a cell or group of cellsas a function of data values. Specifying background color andopacity/transparency can be done similarly with the case for color ofthe font is determined, as illustrated in FIG. 39. For example, a usercan affect the control the background color and opacity/transparencywith the sliders or by simply clicking on a region of color or grayscalechart. The background color control can also be incorporateauto-scaling. Optionally, and the color can be determined depending onthe ratio of the value of the cell to the values in the column.

Similarly, the invention provides for the background texture or stipplesof a cell or group of cells can also be varied as a function of datavalues. FIG. 44 depicts an example use of controlling the backgroundtexture or stipples of cells as a function of data values in aspreadsheet or other form of tabular data. FIG. 45 depicts an exemplaryinterface that can be used to specify the background texture or stipplesto be applied to a cell or a group of cells. The stipples of a cell canbe made up of dots, shapes, or grid lines. The size, thickness, density,or opacity of the background can be adjusted with sliders as well.Stipples can be used alone or in combination with the background color.

The invention provides for the specifying the location of a displayeddata element within a cell. FIG. 46 depicts an exemplary interface thatcan be used to specify the position of text being displayed in a cell.Users can move sliders by clicking and dragging on the icon and movingit vertically or horizontally to specify vertical or horizontal positionof text displayed in a cell. Users can specify the point of vertical andhorizontal centering by clicking on a region in the provided box andselecting how texts are aligned horizontally or vertically relative tothe point specified. In an embodiment, values representing theleft-right or up-down ratio can be simply entered in text boxes by theusers.

The invention provides for the specifying the location of a displayedrotation of text, symbols, or other information within a cell. FIG. 47depicts an exemplary interface that can be used to specify the rotationof text, symbols, or other information displayed combined with thevertical or horizontal placement of the text. By clicking and draggingthe right tip of the slider handle, the user can control the text to berotated counter-clockwise, and vice versa. Vertical or horizontal shiftof the position of the text can be controlled by clicking and draggingthe middle of the slider handle to the desired direction.

The invention provides for a user interface to select of the color of atleast one border of a cell, the thickness of at least one border of acell enveloping a font, the thickness of at least one border outliningof a plurality of contiguous cells, and for a rendered image element.

The invention provides for data of a specified category to be extractedfrom cells and matched with hidden formula to obtain data used in thevisualization. Data from another associated group of cells can be usedfor another attribute in the visualization. For example, data from onecategory can be used to determine the type of object displayed, anothercategory can be used to determine the location of the object displayed,another category can be used to determine the number of objectsdisplayed, and data from another category can be used to determine thesize of the object(s) displayed. Existing spreadsheet applications offervarious types of charts, but the placing of the objects are limited toaligning them in certain orders or directions.

The invention provides for visualizations to be co-rendered with thespreadsheet program. The invention provides for visualizations displayedon a specified region of a spreadsheet. In an embodiment, the inventionprovides for the result of visualizations to be exported to outside ofthe spreadsheet program.

Spreadsheet Visualizations

The invention provides for visual aspects of groups of cells to bevaried together as a function of a data value. For example, FIG. 48depicts an exemplary visualization wherein an attribute of a column areaof data elements is varied as a group. In this example a set ofcollected data containing the state, main industry, population, theincrease in population, number in current work force, unemployment rate,and the increase in unemployment rate can be used for differentpurposes. Depending on what category the emphasis is on, an attribute ofthat column can be varied. Such emphasis can be done by varying the fontboldness, font size, or font color. FIG. 49 depicts an exemplaryvisualization wherein the shading of a row area of data elements isvaried as a group as a function of a data value and also wherein thefont boldness of a column area of data elements is varied as a group asa function of another data value.

The invention provides for a reordering of tabular rows and/or columnsas a function of data values. Unlike a traditional spreadsheet datasort, such a reordering operates as a “live” function, responsive tointeractive changes of the underlying data values. FIG. 50 illustrates amodel of a priority queue wherein the rows can automatically reorderamong themselves as underlying priority sales change. Jobs to be donecan have different attributes, such as location or types of jobs alongwith numbers that indicate the rating of priority. A user would expectjobs with a higher priority rating to be displayed in a higher positionof the priority queue tree listing and to reorder as calculated priorityvalues change.

In another aspect of the invention function or mapping utilities can beincluded in the program. FIG. 51A illustrates a basic map-basedvisualization example generated from previous data. FIG. 51B illustratesthe visualization of FIG. 51A augmented with additional spatiallylocated symbols or parameterized glyphs.

Interactive Tabular Spreadsheet Metaphor for Presentation ofVisualization Renderings

The invention provides for interactive tabular spreadsheet metaphor forpresentation of visualization renderings. A cell may be selected and thedata used to create the enclosed visualization can be captured (asstructured for the enclosed visualization) to be used in a subsequentvisualization or calculation. A cell may be selected and thepre-visualization dataflow and processing steps used to create theenclosed visualization can be captured (as structured for the enclosedvisualization) to be used in a subsequent visualization or calculation.A cell may be selected and the visualization format employed by theenclosed visualization can be captured (as structured for the enclosedvisualization) to be used in a subsequent visualization or calculation.

Two cells can be selected and the data used to create the enclosedvisualizations can be captured (as structured for the enclosedvisualization) and provided to a mathematical operation (for example,adding, subtracting, multiplying, convolving, etc.) to form a newdataset to be used in a subsequent visualization or calculation.Similarly, multiple cells can be selected and the data used to createthe enclosed visualizations can be captured (as structured for theenclosed visualization) and merged to form a new dataset to be used in asubsequent visualization or calculation. Embodiments of the inventioncan also include one or more of the following optional features:

-   -   At least part of the captured data can be re-sampled. In an        embodiment, redundant data occurring in the merged dataset is        removed;    -   Gaps in data occurring in the merged dataset are noted. In an        embodiment, gaps in data occurring in the merged dataset can be        filled in via interpolating functions;    -   At least one partial row and at least one partial column can be        selected and an operation applied to the underlying data;    -   At least one partial row and at least one partial column can be        selected and an operation applied to the individually enclosed        visualizations;    -   At least one partial row and at least one partial column can be        selected and at least one attribute and/or parameter of a common        visualization operation can be adjusted and applied to each of        the individually enclosed visualizations;    -   At least one partial row and at least one partial column can be        selected and at least one attribute and/or parameter of a common        pre-visualization operation can be adjusted and applied to each        of the individually enclosed visualizations;    -   At least one partial row and at least one partial column can be        selected and at least one attribute and/or parameter of a common        data source selection can be adjusted and applied to each of the        individually enclosed visualizations.

Exemplary 3D Graphics Augmentations to Spreadsheet Visualizations

The invention provides for the inclusion of useful novel 3D graphicsvisualization functions for multidimensional data representation, dataquery, and numerical solution tools for simultaneous interactivenumerical equations. These can be incorporated in various ways, asdescribed below, to expand spreadsheet—based visualization capabilitiesyet further.

A first set of the useful novel 3D graphics visualization functionsprovided for by the invention pertain to multidimensional datavisualizations based on 2D-surfaces embedded in a 3D visual field. FIG.52 data or spreadsheet, these separated by a vertical gap within a 3Dvisual field as provided for by the invention. The invention alsoprovides for interactively shifting the observation point with respectto these objects for more detailed feature, theme, or trend inspection.The viewpoint can be changed according to the 6 degrees of freedom(three translations, three angles) of rigid motion. FIG. 53 depicts withdashed lines a few ways in which the observation viewpoint can be movedwith respect to these abstract objects. The viewpoint is changed underthe control of an HDTP or can be changed under the control of anadvanced mouse.

The invention also provides for moving the 2D-surface representing a 3Ddata plot within the 3D visual field. The 2D-surface can be movedaccording to the 6 degrees of freedom (three translations, three angles)of rigid motion. FIG. 54 depicts a translation and rotation of a 3D dataplot within the 3D visual field. The 3D data plot can be moved under thecontrol of an HDTP or the 3D data plot can be moved under the control ofan advanced mouse.

The 2D-surface representing a 3D data plot within the 3D visual fieldcan be virtually illuminated by one or more lighting sources. The one ormore lighting sources can be moved according to the 6 degrees of freedom(three translations, three angles) of rigid motion. The one or morelighting sources can also permit control of the color and intensity ofvirtual light emitted. A lighting source can be moved and controlledunder the control of an HDTP or under the control of an advanced mouse.

The data plotted in the 2D-surface representing a 3D data plot withinthe 3D visual field can directly echo the data displayed in theexemplary planar array of associated tabular data or spreadsheet and/orcan originate from another set of tabular data or spreadsheet regionand/or can originate from other data. This is suggested by FIG. 55A. Theinvention also provides for processing of the data via mathematicaltransformations, statistical processing, signal processing, etc. priorto creation of the 2D-surface representing at least 3-dimensional data.This is suggested by FIG. 55B. In an embodiment, the 2D-surfacerepresenting at least 3-dimensional data can plot filtered or averagedversions of tabular data, spreadsheet data, or other data. In anembodiment, and the filtering or averaging can be controlled by aninteractive parameter.

The invention provides for 2D-surface representing of at least3-dimensional data to be rendered along with a height measuring visualso that the local color of the surface need not be a function of thevertical value being plotted. This frees up the local 2D-surface colorto be used to represent an additional dimension of data. FIG. 56illustrates an exemplary height measuring visual in its use with2D-surface representing at least 3-dimensional data and useful forrepresenting at least 4-dimensional data.

The invention provides for a 2D-surface representing at least3-dimensional data to be rendered with a parameterized locally varyingsurface texture that can be used to represent an additional dimension ofdata, with numerical values rendered on the surface that can be used torepresent an additional dimension of data, with symbols rendered on thesurface that can be used to represent an additional dimension of data,or with parameterized glyphs rendered on the surface that can be used torepresent (an) additional dimension(s) of data.

The invention provides for a slicing function to provide level set data.FIG. 57 depicts an exemplary slicing function, here used to provide a(planar-slice) level set curve. Other uses, implementations, andapplications for slicing functions are also possible.

The invention provides for a more general surface intersection tool thatcan be used as a numerical solution operator for plotted, interpolated,and processed data. FIG. 58A illustrates two exemplary 2D-surfaces, eachrepresenting at least 3-dimensional data. Each of these can beinteractively moved and rotated as described earlier. FIG. 58Billustrates the interactively created intersection of the two exemplary2D-surfaces and the resultant intersection curve. A 2D-surfaceintersection tool allows the numerical values of selected points on acurve formed by the intersecting surfaces to be numerically calculatedand visually displayed. The numerical values can also be used for otherpurposes and other subsequent calculations.

The sample points can be selected according to the details of theintersection. An adaptive sampling method can be employed. The resultantintersection data can be interpolated and re-sampled according toanother sampling strategy. More than two surfaces can be intersectedsimultaneously, and various pair wise and group intersection data can becaptured and displayed.

A second set of the useful novel 3D graphics visualization functionsprovided for by the invention pertain to multidimensional datavisualizations based on 1D-curves embedded in a 3D visual field.

The invention provides for a curve to be generated by tabular data andto be suspended over tabular data used to create it or otherwiseassociated with the curve. The invention provides for the curve to bevisually linked to a geometric rendering of tabular data. These featuresare illustrated by FIG. 59. The invention provides for the curve to begiven a thickness and or color under the control of one or more datavalues. The invention provides for the color of the curve to be variedas a function of tabular (or other) data values. These features areillustrated by FIG. 60. The invention provides for the thickness of thecurve to be varied as a function of tabular (or other) data values. Thisfeature is illustrated in FIG. 61.

Such curve plotting utilities can be provided with functions, features,and operations like that of the 2D-surface utilities described above.The invention also provides for interactively shifting the observationpoint with respect to these objects for more detailed feature, theme, ortrend inspection. The viewpoint can be changed according to the 6degrees of freedom (three translations, three angles) of rigid motion.The viewpoint is changed under the control of an HDTP or an advancedmouse. The invention also provides for moving a curve within the 3Dvisual field. The curve can be moved according to the 6 degrees offreedom (three translations, three angles) of rigid motion. The2D-surface can be moved under the control of an HDTP or an advancedmouse.

A curve of any thickness can be treated as if it comprises a surfacethat can be virtually illuminated by one or more lighting sources. Theone or more lighting sources can be moved according to the 6 degrees offreedom (three translations, three angles) of rigid motion. The one ormore lighting sources can also permit control of the color and intensityof virtual light emitted. In an embodiment, a lighting source can bemoved and/or controlled under the control of an HDTP or an advancedmouse.

The data plotted in a curve can directly echo the data displayed in theexemplary planar array of associated tabular data or spreadsheet and/orcan originate from another set of tabular data or spreadsheet regionand/or can originate from other data as suggested earlier in FIG. 55A.The invention also provides for processing of the data via mathematicaltransformations, statistical processing, signal processing, etc. priorto creation of the curve as was suggested earlier in FIG. 55B. A curvecan plot filtered or averaged versions of tabular data, spreadsheetdata, or other data. In an embodiment, the filtering or averaging can becontrolled by an interactive parameter.

The invention provides for a curve to be rendered along with a heightmeasuring visual, and also provides for a locally variable parameterizedtexture to be imposed on the curve and for the texture to be varied as afunction of tabular or other data.

The invention provides for the curve to be rendered with numericalvalues displayed adjacently at specific points on the curve and thatthese can be used to represent an additional dimension of data. Theinvention also provides for the curve to be rendered with symbolsdisplayed adjacently at specific points on the curve and that these canbe used to represent an additional dimension of data. The inventionfurther provides for the curve to be rendered with parameterized glyphsdisplayed adjacently at specific points on the curve and that these canbe used to represent (an) additional dimension(s) of data.

The invention provides for a curve intersection tool that can be used asa numerical solution operator for plotted, interpolated, and processeddata. FIG. 62 illustrates two exemplary curves, each representing atleast 2-dimensional data. Each of these can be interactively moved androtated as described earlier, interactively creating an intersection ofthe two exemplary curves and at least one resultant intersection point.In an embodiment, a 2D-surface intersection tool allows the numericalvalues of intersection points to be numerically calculated and visuallysignified. The numerical values can also be used for other purposes andother subsequent calculations.

In an embodiment, more than two curves can be intersectedsimultaneously, and various pair-wise and group intersection data can becaptured and displayed.

For example, a curve and a surface can be intersected simultaneously,and intersection data can be captured and displayed. As another example,more than two curves and at least one surface can be intersectedsimultaneously, and various pair-wise and group intersection data can becaptured and displayed.

While the invention has been described in detail with reference todisclosed embodiments, various modifications within the scope of theinvention will be apparent to those of ordinary skill in thistechnological field. It is to be appreciated that features describedwith respect to one embodiment typically can be applied to otherembodiments.

The invention can be embodied in other specific forms without departingfrom the spirit or essential characteristics thereof. The presentembodiments are therefore to be considered in all respects asillustrative and not restrictive, the scope of the invention beingindicated by the appended claims rather than by the foregoingdescription, and all changes which come within the meaning and range ofequivalency of the claims are therefore intended to be embraced therein.Therefore, the invention properly is to be construed with reference tothe claims.

We claim:
 1. A method for visualizing tabular data of an electronicspreadsheet, the method comprising: obtaining data values from tabulardata, the tabular data comprising plurality of data values organized asan array; rendering a visualization of a one-dimensional curve, thevisualization computed from the data values obtained from tabular data,the rendering further computed from position data, orientation data, andviewpoint data, the visualization rendering a view of the curve in a3-dimensional space in a visual display area of a computer screen;wherein the visualization is displayed together with a display of thetabular data of the electronic spreadsheet, and wherein thevisualization of the rendered curve further comprises a varying color,the color representing a value responsive to the tabular data.
 2. Themethod of claim 1, wherein the position data is received from a userinterface.
 3. The method of claim 1, wherein the orientation data isreceived from a user interface.
 4. The method of claim 1, wherein theviewpoint data is received from a user interface.
 5. The method of claim1, wherein the rendered visualization is supplemented with at least onesymbol rendered adjacent to the curve, the symbol representing a valueresponsive to the tabular data.
 6. The method of claim 1, wherein therendered visualization is supplemented with at least one parameterizedglyph rendered adjacent to the curve, the parameterized glyphrepresenting a value responsive to the tabular data.
 7. The method ofclaim 1, the method further comprises rendering a visual heightmeasuring tool in the visualization.
 8. The method of claim 1, whereinthe one-dimensional curve represents filtered versions of the tabulardata.
 9. The method of claim 1, wherein the visualization is renderedwith numerical values displayed adjacently at specific points on thecurve, the displayed numerical values representing an additionaldimension of data.
 10. The method of claim 1, wherein theone-dimensional curve represents statistically processed of the tabulardata.
 11. The method of claim 1, wherein the visualization additionallycomprises visual links to tabular data items within a geometricrendering of tabular data.
 12. A method for visualizing tabular data ofan electronic spreadsheet, the method comprising: obtaining data valuesfrom tabular data, the tabular data comprising plurality of data valuesorganized as an array; rendering a visualization of a one-dimensionalcurve, the visualization computed from the data values obtained fromtabular data, the rendering further computed from position data,orientation data, and viewpoint data, the visualization rendering a viewof the curve in a 3-dimensional space in a visual display area of acomputer screen; wherein the visualization is displayed together with adisplay of the tabular data of the electronic spreadsheet, and whereinthe visualization of the rendered curve further comprises a varyingthickness, the thickness representing a value responsive to the tabulardata.
 13. The method of claim 12, wherein the position data is receivedfrom a user interface.
 14. The method of claim 12, wherein theorientation data is received from a user interface.
 15. The method ofclaim 12, wherein the viewpoint data is received from a user interface.16. The method of claim 12, wherein the rendered visualization issupplemented with at least one symbol rendered adjacent to the curve,the symbol representing a value responsive to the tabular data.
 17. Themethod of claim 12, wherein the rendered visualization is]supplementedwith at least one parameterized glyph rendered adjacent to the curve,the parameterized glyph representing a value responsive to the tabulardata.
 18. The method of claim 12, the method further comprises renderinga visual height measuring tool in the visualization.
 19. The method ofclaim 12, wherein the one-dimensional curve represents filtered versionsof the tabular data.
 20. The method of claim 12, wherein thevisualization is rendered with numerical values displayed adjacently atspecific points on the curve, the displayed numerical valuesrepresenting an additional dimension of data.
 21. The method of claim12, wherein the one-dimensional curve represents statistically processedof the tabular data.
 22. The method of claim 12, wherein thevisualization additionally comprises visual links to tabular data itemswithin a geometric rendering of tabular data.